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Figure  1:  Units  of  Assessment  Represented 
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Annex  A  -  Information  on  Participants 


Table  1:  Institutions  Represented 

Aberdeen  University 
Anglia  Polytechnic  University 
Bath  University 
Bournemouth  University 
Bristol  University 
Brunei  University 
Cambridge  University 
Cardiff  University 
Chester  College 
Cranfield  University 
Durham  University 

Edge  Hill  College  of  Higher  Education 
Edinburgh  College  of  Art 
Exeter  University 
Glasgow  Caledonian  University 
Glasgow  University 
Heriot  Watt  University 
Imperial  College 
Kings  College  London 
Lancaster  University 
Liverpool  John  Moores  University 
Liverpool  University 
London  Metropolitan  University 
Luton  University 
Manchester  University 
Napier  University 
Newcastle  University 
Northumbria  University 
Oxford  University 
Plymouth  University 
Portsmouth  University 
Queen  Mary's  School  of  Medicine  and 
Dentistry 

Queen's  University  Belfast 
Reading  University 


Royal  Holloway  University  of  London 

Salford  University 

Southampton  University 

Southbank  University 

St  Martin’s  College 

Sussex  University 

Thames  Valley  University 

The  Bolton  Institute 

UMIST 

University  of  Central  Lancashire 
University  College  London 
University  of  East  Anglia 
University  of  Glamorgan 
University  of  Kent  at  Canterbury 
University  of  St  Andrews 
University  of  Sunderland 
University  of  Surrey 
University  of  Teeside 
University  of  the  West  of  England 
University  of  Ulster 
University  of  Wales,  Aberystwyth 
University  of  Wales  Institute  Cardiff 
University  of  Wales  Newport 
University  of  Wales  Swansea 
University  of  Wales  Bangor 


Table  2:  Unrepresented  Units  of  Assessment 


UoA  No. 

1 

Clinical  Laboratory  Sciences 

4 

Clinical  Dentistry 

5 

Pre-clinical  Studies 

8 

Pharmacology 

17 

Veterinary  Science 

22 

Pure  Mathematics 

23 

Applied  Mathematics 

29 

Electrical  and  Electronic  Engineering 

31 

Mineral  and  Mining  Engineering 

34 

Town  and  Country  Planning 

41 

Social  Work 

44 

Accounting  and  Finance 

45 

American  Studies 

46 

Middle  Eastern  and  African  Studies 

47 

Asian  Studies 

52 

German,  Dutch  and  Scandinavian  Languages 

53 

Italian 

54 

Russian,  Slavonic  and  East  European  Languages 

55 

Iberian  and  Latin  American  Languages 

57 

Classics,  Ancient  History,  Byzantine  and  Modern  Greek 

58 

Archaeology 

62 

Philosophy 

63 

Theology,  Divinity  and  Religious  Studies 

65 

Communication,  Cultural  and  Media  Studies 

66 

Drama,  Dance  and  Performing  Arts 

69 

Sports-related  Subjects 

Table  3:  Participants  by  Role  and  Gender 


Role 

Total 

Admin 

Lecturer 

Research 

Senior 

Gender  female 

16 

8 

1 

13 

38 

male 

20 

11 

8 

65 

104 

Total 

36 

19 

9 

78 

142 

Table  4:  Research  and  Teaching  Staff  by  Field  and  Role 


Field 

Total 

Arts 

Medicine 

Social 

Role  Lecturer' 

3 

10 

6 

19 

Research1 2 

1 

8 

0 

9 

Senior3 

18 

40 

20 

78 

Total 

22 

58 

26 

106 

1  Lecturer  or  Senior  Lecturers  with  teaching  and  research  responsibilities  (principal  lecturer,  senior  lecturer, 
lecturer) 

2  Research  fellows  with  near  exclusive  research  responsibilities 

3  Senior  academics  with  large  management  responsibility  (professor,  head  of  department,  reader) 


3 


Annex  B  -  Task  1  Results 

Two  participants  had  to  be  excluded  from  the  analysis  because  of  a  clerical  error,  and  one  participant  did 
not  vote.  It  should  also  be  noted  that  a  few  participants  did  not  cast  all  five  of  their  votes. 

Table  1:  Characteristics  of  High  Quality  Research 


Cluster 

Characteristic 

Workshop 

Votes 

Academic  Impact 

Academic  Impact 

4 

11 

Academic  Impact 

Academic  Impact 

5 

10 

Academic  Impact 

Academically  influential 

6 

1 

Academic  impact 

Potential  to  have  an  impact 

8 

6 

Collaborative 

Collaboration 

3 

1 

Collaborative 

Themed  and  collaborative 

6 

2 

Creative 

Creativity 

5 

4 

Creative 

Creative 

6 

7 

Creative 

Creativity 

8 

8 

Defining  Research  Agenda 

Potential  to  move  discipline  forward 

1 

15 

Defining  Research  Agenda 

Highlights  new  questions 

1 

8 

Defining  Research  Agenda 

Moving  the  field  forward 

2 

15 

Defining  Research  Agenda 

Agenda  setting 

4 

9 

Defining  Research  Agenda 

Changing  thinking 

4 

2 

Defining  Research  Agenda 

Defining  an  area 

5 

3 

Defining  Research  Agenda 

Defines  research  agenda 

6 

13 

Defining  Research  Agenda 

Creates  debate 

6 

3 

Defining  Research  Agenda 

Advancement  of  field 

7 

5 

Defining  Research  Agenda 

Taking  discipline  forward 

7 

4 

Defining  Research  Agenda 

Agenda  challenging 

8 

4 

Defining  Research  Agenda 

Stimulates  further  development 

8 

3 

Defining  Research  Agenda 

Developing  new  agenda 

8 

2 

Defining  Research  Agenda 

Agenda  setting 

9 

13 

High  Quality  Publication 

High  impact  publication 

3 

6 

High  Quality  Publication 

Publication  output 

4 

9 

High  Quality  Publication 

High  quality  publication 

5 

9 

High  Quality  Publication 

Output  in  terms  of  publication 

7 

8 

Innovation 

Innovative 

1 

8 

Innovation 

Innovation 

3 

6 

Innovation 

Innovation 

4 

7 

Innovation 

Innovation  and  boundary  pushing 

8 

2 

Innovation 

Innovative 

9 

10 

International  Recognition 

National  and  international  impact 

1 

8 

International  Recognition 

International  Impact 

3 

6 

International  Recognition 

International  recognition 

4 

15 

International  Recognition 

Internationally  competitive 

5 

6 

International  Recognition 

International  recognition 

6 

11 

International  Recognition 

Internationally  leading 

6 

2 

International  Recognition 

International  impact 

7 

5 

International  Recognition 

International  peer  recognition 

8 

9 

International  Recognition 

National  and  internationally  leading 

9 

7 

Long  Lasting 

Long  lasting 

2 

6 

Long  Lasting 

Lasting  impact 

3 

4 

Long  Lasting 

Longevity 

4 

5 

Long  Lasting 

Long  lasting 

6 

3 

Long  Lasting 

Longevity 

7 

3 

Long  Lasting 

Longevity 

9 

4 

4 


Cluster 

Original 

Original 

Original 

Original 

Original 

Original 

Original 

Original 

Original 

Peer  Recognition 

Peer  Recognition 

Peer  Recognition 

Peer  Recognition 

Peer  Recognition 

Relevance 

Relevance 

Relevance 

Relevance 

Relevance 

Research  Culture 

Research  Culture 

Rigour 

Rigour 

Rigour 

Rigour 

Rigour 

Rigour 

Rigour 

Rigour 

Scholarship 

Scholarship 

Scholarship 

Utility 

Utility 

Utility 

Utility 

Utility 

Utility 

Value  for  Money 
Value  for  Money 
Value  for  Money 
Value  for  Money 
Value  for  Money 


Characteristic 

Originality 
Originality 4 
Original  findings 
Originality  of  method 
Originality  and  innovation 
Originality  and  novelty 
Original 

Novel  application 
Original 

Peer  recognition 
Peer/expert  recognition 
Peer  recognition 
Peer  recognition 
Peer  recogniti  on 
Relevance 
Fitness  for  purpose 

Relevant  to  and  integrated  with  teaching 

Public  relevance 

Empirical  relevance 

Research  ethos 

Research  culture 

Rigorous 

Depth 

Rigour 

Methodological  rigour 
Rigorous 

Methodological  rigour 
Rigour  (methods  and  approach) 

Methodological  rigour 

Scholarship,  adding  to  body  of  knowledge 

Scholarship  and  original  synthesis 

Advance  body  of  knowledge 

High  value  for  users 

Applicability 

Utilization  by  end  users 
Impact  on  end  Users 
End  user  relevance 
Professionally  useful 
Productivity 
Valuable 
Added  value 
Value  for  money 
Value  for  money 
Well  informed  (engaged) 

Impact  on  quality  of  experience  (human  and  other) 

Multidisciplinary  and  collaborative 

Impact  on  society 

Leadership  in  the  field 

Authoritative 

Integrity 

Research  income  to  subject  area 


Workshop 

1 

2 

3 

3 

5 

6 

7 

8 
9 
1 
2 

4 

5 
9 

4 

5 

6 
8 
8 

4 

5 
2 
2 
3 

5 

6 

7 

8 
9 
2 
8 
9 
2 
2 

3 
6 
8 
9 
2 
2 

4 

5 

6 
9 
9 
9 
7 
7 
7 
7 
7 


Votes 

7 

0 

7 
2 
15 
14 

8 
2 
6 
4 
14 

14 

15 

4 
2 
3 
1 
2 
1 
1 
7 
15 
9 

5 

6 
9 
11 
5 
11 
12 
7 
21 
9 
5 

3 
11 

4 

5 
3 
2 

6 
6 

3 
7 
1 
1 

12 

11 

7 

4 
2 


4  This  sticky  hexagon  fell  off  the  chart  and  was  not  available  for  voting. 


Cluster 


Characteristic 
Theory  based 
Accurate 
External  funding 
Accessibility 
Deliverable  output 
Quality  at  what  level? 

Regional  and  national  impact 
Holistic  (judge  team) 

Good  project  management 
Known  in  non- monolingual  context 
Wealth  creating 
Coherence 

Subject  to  transparent  process  at  proposal  stage 

Academic  assessment 

Breadth  of  impact 

Number  of  disciples/students 

Inspirational 

Contribution  to  knowledge  and  understanding 
Puzzle  driven 


Workshop 

7 

7 
4 

4 

5 
5 

5 
3 
3 
2 
2 
1 
1 
1 

6 
6 
6 

8 
8 


Votes 

0 

0 

7 

2 

4 

1 

1 

1 

1 

2 

2 

0 

0 

0 

13 

1 

1 

13 

2 


Table  2:  Characteristics  of  Research  Assessment  Systems 


Cluster 

Characteristic 

Workshop 

Accountable 

Accountability 

6 

Accountable 

Accountability 

8 

Appropriate 

Subject  specific 

2 

Appropriate 

Fit  for  purpose 

2 

Appropriate 

Appropriate  for  discipline 

5 

Appropriate 

Sensitive  to  changes  in  discipline  boundaries 

6 

Appropriate 

Sensitivity  to  discipline 

7 

Appropriate 

Appropriate  to  expertise 

8 

Appropriate 

Appropriateness  of  criteria  for  discipline 

8 

Appropriate 

Appropriate  method  within  discipline 

9 

Career  Sensitive 

Supportive  of  early  career 

7 

Career  Sensitive 

Supporting  careers 

9 

Career  Sensitive 

Sensitive  to  life  changes 

9 

Comparable 

Comparability  across  subjects 

2 

Comparable 

Cross  subject  comparability 

4 

Comparable 

Comparable  and  suitable 

5 

Comparable 

Consistency  across  disciplines 

6 

Comparable 

Parity  of  criteria  within  clusters 

8 

Comparable 

Comparability  of  clusters 

8 

Comparable 

Comparable  across  disciplines 

9 

Cost  efficient 

Cost  effective 

4 

Cost  efficient 

Cost  efficient 

5 

Cost  efficient 

Cost  effective 

6 

Cost  efficient 

Efficiency 

6 

Cost  efficient 

Cost  effective 

7 

Cost  efficient 

Efficient 

9 

Credible 

Broadly  acceptable 

4 

Credible 

Credibility 

5 

Explicit 

Clear  rules 

4 

Votes 

1 

4 

9 

2 

9 

4 
13 
9 

5 
0 

4 
0 
0 

9 

10 
9 
2 

5 
1 
7 
1 
5 

3 
1 
1 
2 

4 
9 
13 
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Cluster 

Characteristic 

Workshop 

Votes 

Explicit 

Clear  and  acceptable  criteria 

5 

12 

Explicit 

Explicit  criteria 

6 

7 

Fair 

Fair  and  flexible 

1 

4 

Fair 

Fairness 

2 

9 

Fair 

Fairness 

4 

10 

Fair 

Fair 

6 

2 

Fair 

Fair 

7 

8 

Fair 

Fair  reward 

8 

7 

Fair 

Fair  to  individual 

8 

4 

Fair 

Fair  to  institutions 

8 

0 

Feedback 

Meaningful  feedback 

4 

3 

Feedback 

Good  feedback 

8 

3 

Feedback 

Feedback 

9 

1 

Flexible 

Flexibility 

3 

5 

Flexible 

Flexibility  across  disciplines 

6 

10 

Flexible 

Flexibility  across  institutes 

6 

2 

Flexible 

Flexible  timing 

8 

1 

Flexible 

Flexible  across  disciplines 

9 

8 

Flexible 

Responsive 

9 

1 

Forward  Looking 

Forward  looking 

6 

3 

Forward  Looking 

Forward  looking 

9 

0 

Improve  Research  Base 

Improve  research  base 

4 

4 

Improve  Research  Base 

Be  able  to  advance  field 

5 

7 

No  game  playing 

No  gaming 

1 

0 

No  game  playing 

No  scope  for  game  playing 

6 

7 

No  game  playing 

Minimise  game  playing 

7 

6 

Not  Burdensome 

Not  burdensome 

1 

6 

Not  Burdensome 

Not  burdensome 

2 

13 

Not  Burdensome 

Not  onerous 

4 

3 

Not  Burdensome 

Low  burden 

7 

6 

Not  Burdensome 

Not  burdensome 

8 

5 

Not  Burdensome 

Not  burdensome 

9 

11 

Objective 

Objectivity 

2 

10 

Objective 

Objectivity 

7 

4 

Objective 

Objective 

9 

7 

Output  is  Assessed 

Dissemination  of  outcomes 

1 

3 

Output  is  Assessed 

Output  is  assessed 

1 

0 

Output  is  Assessed 

Biased  towards  output  measures 

6 

7 

Output  is  assessed 

Assess  output  and  not  input 

7 

3 

Peer  Recognition 

Peer  review 

1 

12 

Peer  Recognition 

Peer  recognition 

3 

11 

Peer  Recognition 

Peer  review 

4 

8 

Peer  Recognition 

Assess  impact  in  peer  group 

5 

5 

Peer  recognition 

Peer  recognition 

7 

1 

Peer  Recognition 

Peer  review 

9 

14 

Quality  not  volume 

Assess  quality  not  volume 

5 

9 

Quality  not  volume 

Measure  quality  not  volume 

6 

17 

Rigour 

Rigorous 

2 

9 

Rigour 

Rigour 

3 

8 

Rigour 

Rigour 

4 

8 

7 


Cluster 

Rigour 

Robust 

Robust 

Transparent 

Transparent 

Transparent 

Transparent 

Transparent 

Transparent 

Transparent 

Transparent 

Transparent 

Transparent 


Characteristic 

Rigour 

Robust 

Robustness 

Transparency 

Transparency 

Transparency 

Transparent  process 

Transparent 

Transparent 

Transparent 

Transparent  process 

Transparency  of  rules 

Transparent 

Consistent 

Inclusive 

Periodic  metrics 

No  spill  over  effects 

Dynamic 

Competence 

Legitimacy 

Consensus 

Match  funding  to  results 

Qualitative 

Holistic 

No  double  counting 
Continuity 
Clear  function 
Regional  sensitivity 
Assessment  impact  on  end  users 
Citation  analysis 

Praxis  (integration  of  theory  and  action) 

Recognition  of  process  as  well  as  outputs  (capacity  building) 
Simplicity 

Clarity  of  outcome  (in  funding) 

Value  different  types  of  output 
External  validation 
Timely  notice  of  criteria 

Differentiate  between  fields  according  to  research  outputs 
Proper  assessment  of  interdisciplinary  research 
Independent  judges 
Useful  output 

Take  into  account  end  users 
Consequences  reflect  quality 
Contribution  from  international  experts 
Reflexive 

Longevity  of  research 
Quantitative  analysis  where  possible 
Systematic 
Multifaceted 

Respect  different  opinions 


Workshop 

9 

1 

3 

1 

2 

3 

4 

5 

6 

7 

8 
8 
9 
1 
1 
1 
1 
2 
2 
2 
2 
2 
3 
3 

3 

4 
4 

4 

5 
5 
5 
5 

5 

6 
6 
6 
6 
7 
7 
7 
7 
7 

7 

8 
8 
8 
9 
9 
9 
9 


Votes 

3 
11 

4 
9 
13 

7 
13 

8 

9 

10 
8 

4 
18 

3 
2 
0 
0 
8 

5 

4 

4 
0 
8 
1 
1 

5 

5 

3 

6 

4 

4 

3 
0 

7 

5 

4 
4 

8 

6 
6 
2 
2 
0 
6 
4 
2 
8 
4 
3 
1 


8 


Cluster 

Characteristic 

Workshop 

Votes 

Look  at  usefulness 

9 

1 

Joined  up  with  other  funders 

9 

1 

Longevity 

9 

0 

Accurate  credit 

9 

0 

Retrospective 

9 

0 

9 


Table  3:  Total  Votes  for  Clustered  Characteristics  of  High  Quality 
Research 

Independent 

Cluster  of  Characteristic  Mentions  Votes 

Defining  Research  Agenda  8  99 

Rigour  7  71 

International  Recognition  8  66 

Original  8  61 

Peer  Recognition  5  50 

Scholarship  3  40 

Utility  5  37 

High  Quality  Publication  4  32 

Innovation  5  31 

Academic  Impact  4  26 

Long  Lasting  6  24 

Value  for  Money  5  20 

Creative  3  19 

Breadth  of  Impact  1  13 

Contribution  to  knowledge  and  understanding  1  13 

Impact  on  Society  1  12 

Leadership  in  the  Field  1  11 

Relevance  4  9 

Research  Culture  2  8 

External  Funding  1  7 

Authoritative  1  7 

Well  informed  (engaged)  1  7 

Deliverable  Output  1  4 

Integrity  14 

Collaborative  2  3 

Known  in  non-monolingual  context  1  2 

Wealth  creating  1  2 

Puzzle  driven  1  2 

Research  income  to  subject  area  1  2 

Good  Project  Management  1  1 

Regional  and  National  Impact  1  1 

Impact  on  quality  of  experience  (human  and  other)  1  1 

Number  of  disciples/students  1  1 

Quality  at  what  level?  1  1 

Multidisciplinary  and  collaborative  1  1 

Inspirational  1  1 

Accessibility  1  1 

Holistic  (judge  team)  11 


Table  4:  Total  Votes  for  Clustered  Characteristics  of  Research 
Assessment  Systems 

Independent 

Cluster  of  Characteristic  Mentions  Votes 

Transparent  9  98 

Appropriate  6  51 

Peer  Recognition  6  50 

Not  Burdensome  6  44 

Fair  6  42 

Comparable  6  41 

Explicit  3  30 


Independent 


Cluster  of  Characteristic 

Mentions 

Votes 

Rigour 

4 

27 

Flexible 

4 

27 

Quality  not  volume 

2 

26 

Objective 

3 

21 

Robust 

2 

15 

Credible 

2 

13 

Cost  efficient 

5 

13 

No  scope  for  game  playing 

3 

13 

Improve  Research  Base 

2 

11 

Output  is  Assessed 

3 

10 

Qualitative 

1 

8 

Differentiate  between  fields  according  to  research  outputs 

1 

8 

Quantities  analysis  where  possible 

1 

8 

Dynamic 

1 

8 

Clarity  of  Outcome  (in  funding) 

1 

7 

Feedback 

3 

7 

Independent  Judges 

1 

6 

Contribution  from  International  Experts 

1 

6 

Assessment  impact  on  end  users 

1 

6 

Proper  assessment  of  interdisciplinary  research 

1 

6 

Value  different  types  of  output 

1 

5 

Competence 

1 

5 

Clear  Function 
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Hybrid  System  2:  Based  on  Expert  Review 

Similar  to  current  system,  but  tweaked. 

Longer  period  between  reviews  (ie  >  5  years). 
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Appropriate  allocation  of  funds,  need  to  take  into  account  both 
individuals  and  departmental  excellence. 


oo 


RA  are  each  individual  forms  that  make  up  a  submission  to  the  RAE.  RA2  (Research  output)  is  a  list  of  up  to  four  items  of  research  output  for  each  researcher.  RA3  (Research  Students  and  Research 
Studentships)  details  the  research  students  and  studentships  held  by  individual  researchers.  RA4  (External  Research  Income)  details  the  external  research  income  awarded  to  researchers  within  the 
submiss, on.KA5  (Textual  description)  is  concerned  with  the  research  strategies,  policies,  structure  and  environment.  RA6  (General  observations  and  additional  information)  relates  to  marks  of  esteem 

attached  to  individual  researchers,  and  any  additional  observations.  It  includes  an  opportunity  for  HEIs  to  comment,  in  confidence,  on  circumstances  which  have  reduced  the  research  activity  of  specific 
individuals  during  the  assessment  period. 


the  department  as  independent  investigators. 
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Lack  of  clarity  on  any  aspects. 
Changing  of  rules  or/ and  outcomes. 


Hybrid  System  14:  Expert  Review 

Came  up  with  framework  in  which  assessment  system  would  operate. 
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Discussion 

There  is  no  cross  disciplinary  comparability.  This  allows  flexibility. 
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Legitimization/justification/ trust  of  funding 
academic  community  with 
academics/ politicians/ public. 

Entire  process  is  zero  sum  game,  there  is  a 
limited  pot  of  money. 
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Discussion 

Fewer  grades?  We  already  have  three  de  facto  grades  in  the  RAE  for  funding  so  might  as  well  recognize  it. 
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